In safety-critical applications, autonomous agents may need to learn in an environment where mistakes can be very costly. In such settings, the agent needs to behave safely not only after but also while learning. To achieve this, existing safe reinforcement learning methods make an agent rely on priors that let it avoid dangerous situations during exploration with high probability, but both the probabilistic guarantees and the smoothness assumptions inherent in the priors are not viable in many scenarios of interest such as autonomous driving. This paper presents an alternative approach inspired by human teaching, where an agent learns under the supervision of an automatic instructor that saves the agent from violating constraints during learning. In this model, we introduce the monitor that neither needs to know how to do well at the task the agent is learning nor needs to know how the environment works. Instead, it has a library of reset controllers that it activates when the agent starts behaving dangerously, preventing it from doing damage. Crucially, the choices of which reset controller to apply in which situation affect the speed of agent learning. Based on observing agents' progress the teacher itself learns a policy for choosing the reset controllers, a curriculum, to optimize the agent's final policy reward. Our experiments use this framework in two environments to induce curricula for safe and efficient learning.

Weaknesses: There were two primary weaknesses that I noticed in the paper: (1) The paper notes that the framework is different from prior curriculum learning work due to learning from prior learners, allowing it to be "data-driven rather than heuristic," but the consequences of that aren't explored. In particular, this may mean that many agents must be trained prior to getting to a good curricular policy, and this may be problematic for practitioners. This fact is somewhat hidden in the experimental evaluations because the evaluation of the optimized sequence of interventions doesn't include the performance of the first 30-100 learners. It would be helpful to either include a clearer argument for the importance of learning the curriculum policy over other approaches or to discuss the possible limitations of needing to learn from many learners (and perhaps the robustness claims would mitigate this limitation to some extent). I would also have liked to see results (perhaps in the appendix) for the sensitivity of the results to the number of learners prior to reaching an "optimized" point.

After the discussion all reviewers support acceptance, noting that the paper lays out a novel, clear, and general framework for safe online RL. This topic is very relevant to the NeurIPS community, and the paper should be disseminated. However, all reviewers expressed at least minor concerns. I strongly encourage the authors to consider this feedback so that they can improve the responses from future readers. In the discussion, it also became clear that a reviewer thought the paper described an agent interacting with *human* students - perhaps future clarifications can avoid this point of confusion.

In safety-critical applications, autonomous agents may need to learn in an environment where mistakes can be very costly. In such settings, the agent needs to behave safely not only after but also while learning. To achieve this, existing safe reinforcement learning methods make an agent rely on priors that let it avoid dangerous situations during exploration with high probability, but both the probabilistic guarantees and the smoothness assumptions inherent in the priors are not viable in many scenarios of interest such as autonomous driving. This paper presents an alternative approach inspired by human teaching, where an agent learns under the supervision of an automatic instructor that saves the agent from violating constraints during learning. In this model, we introduce the monitor that neither needs to know how to do well at the task the agent is learning nor needs to know how the environment works.